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Abstract 

There is no doubt that having evaluation policies makes the evaluator's job 
easier, by providing more transparency to the evaluation process and more 
security to those involved. However, this can only happen when policies are 
presented to potential stakeholders in clear language, as they are 
disseminated and utilized for guiding practice. This suggests that building 
an evaluation culture is necessary in order to effectively implement an 
evaluation policy, so it can be fully utilized. This paper briefly introduces and 
presents what the authors understand as the concept of "evaluation 
culture" and "evaluation policy". Then, it discusses the importance of 
evaluation policies for practice of both evaluation and meta-evaluation, and 
points out the possible consequences of absence of policies. 
Recommendations are derived from these considerations in order to inspire 
procedures to build and implement evaluation policies. Finally, reflections 
are presented based on the Brazilian experience about "culture-policy- 
practice" inter-relations in evaluation. 

Keywords: Evaluation. Evaluation policies. Evaluation culture. Practice. 
Brazilian experience. 
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Cultura de Avaliacao e Politica de Avaliacao 
como guias para a sua Pratica: Reflexoes a 
respeito da Experiencia Brasileira. 
Resumo 

Nao ha duvida de que as politicas de avaliacao tornam o trabalho do avaliador 
uma tarefa mais facil por darem mais transparencia ao processo avaliativo e mais 
seguranca as pessoas envolvidas. No entanto, isso so se da quando essas politicas 
sao apresentadas a stakeholders (interessados) em potencial, em uma linguagem 
clara, ja que elas sao divulgadas e utilizadas para guiar a pratica. Sugere-se, assim, 
que a construcao de uma cultura de avaliacao e necessaria para que uma politica de 
avaliacao seja implementada efetivamente de modo a ser plenamente utilizada. Esse 
artigo apresenta, de maneira concisa, o que os autores entendem por "cultura de 
avaliacao" e "politica de avaliacao"; discute a importancia de politicas de avaliacao 
na pratica da avaliacao e da meta-avaliacao; e destaca as possiveis consequencias da 
ausencia dessas politicas. A partir de tais consideracoes, surgem recomendacoes 
que tern como intuito inspirar o desenvolvimento e a implementacao dessas politi- 
cas. Finalmente, sao apresentadas reflexoes baseadas na experiencia brasileira com 
relacao as inter-relacoes "cultura-politica-pratica" na avaliacao. 

Palavras-chave: Avaliacao. Politicas de avaliacao. Cultura de avaliacao. Pratica. 
Experiencia brasileira. 

Resumen 

Cultura de Evaluation y Politica de 
Evaluation como Guias para su Practica: 
Reflexiones sobre la Experiencia Brasilena. 

No hay duda de que las politicas de evaluacidn toman el trabajo del evaluador 
una tarea mas facil por dar mas transparencia al proceso de evaluacidn y mas 
seguridad a las persona s participantes. Todavia, eso solo se da cuando esas 
politicas son presentadas a stakeholders (interesados) en potencial, utilizando 
lenguaje clara, ya que ellas son divulgadas y utilizadas para guiar la practica. 
Se sugiere, asi, que la construccidn de una cultura de evaluacidn es necesaria 
para que una politica de evaluacidn sea implementada efectivamente para ser 
plenamente utilizada. Ese articulo presenta, de modo conciso, lo que los autores 
entienden por "cultura de evaluacidn" y "politica de evaluacidn"; discute la 
importancia de politicas de evaluacidn en la practica de la evaluacidn y de la 
meta-evaluacidn; y destaca las posibles consecuencias de la ausencia de esas 
politicas. A partir de tales consideraciones, surgen recomendaciones que tienen 
como intuito inspirar el desarrollo y la implementacidn de esas politicas. 
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Finalmente, son presentadas reflexiones basadas en la experiencia brasilena 
considerando las interrelaciones "cultura-politica-practica" en la evaluacion. 
Palavras clave: Evaluacion. Poli'ticas de evaluacion. Cultura de evaluacion. 
Practica. Experiencia brasilena. 

Introduction 

Brazil has had a significant increase in the number of evaluations, especially in 
the educational and the social arena since the beginning of the 90's due to demand 
from the government and the Third Sector. But this growth occurred without 
previous planning. The combination of intense demand with lack of proper plan- 
ning generated several challenges: a lack of professionals in the evaluation field; 
systems, organizations, institutions and people not prepared to be evaluated and to 
use the results of evaluations; lack of a network among the evaluators and those 
interested in evaluation, to stimulate collective development; scarce evaluation lite- 
rature in Portuguese and lack of training courses in order to adequately prepare 
professional evaluators. Consequently evaluations of diverse natures were produ- 
ced which can be grouped into three categories, as related to evaluation policy: (1) 
evaluation without a guiding policy; (2) evaluation with a pre-determined policy 
designed by a small group, not submitted for wide and substantial discussion, nor 
using language accessible to specific groups, often lacking transparency and wide 
dissemination; (3) evaluation with a pre-determined policy designed in an open and 
collective manner, and widely disseminated. 

Discussion needs to be promoted about the strengths and weaknesses of these 
three categories as well as the consequences of not having an evaluation culture 
within society in general. Debate is also needed over the concepts of evaluation 
policy and evaluation culture in order to achieve convergence toward significant 
progress. Careful consideration of these aspects is of utmost importance for the 
democratic development of public policy in Brazil. The country is currently under- 
going a phase when evaluators and evaluees argue about several issues: evaluation 
quality and the effect of evaluation upon communities; the importance of cons- 
tructing an evaluation culture; and to what extent evaluations have contributed to 
the growth of stakeholder's social capital. 

This turmoil of many doubts and few certainties, where policies attempt to 
guide practice, while practice incites policies, in fact offers a precious opportunity 
for one to observe and study the seeds - seeds that are mindlessly and unintentio- 
nally thrown in fertile soil, that nevertheless sprout beautifully, even without pro- 
per care. One must not ignore the outcomes that likely have been produced. Les- 
sons must be learned throughout about the way to improve the processes of "so- 
wing of the seeds" of evaluation policy and the "consumption" of evaluation prac- 
tice. Brazil characterizes the Azure Jay syndrome - a bird from southern Brazil 



Ensaio: aval. pol. publ. Educ, Rio de Janeiro, v. 17, n. 62, p. 169-180, jan./mar. 2009 



172 



Thereza Penna Firme, Ana Carolina Letichevsky, 
Angela Cristina Dannemann e Vathsala Stone 



which, while feeding on the "araucaria" cones (pine seeds), seeks to hide it for 
future consumption, thus unintentionally cultivating woods of "araucaria" pine trees, 
a rare and valuable wood nowadays, for the benefit of mankind. 

Evaluation culture and evaluation policy 

Concept 

Culture can be understood as common beliefs and practices of a group of 
people, in other words, the integrated pattern of human knowledge, belief, and 
behavior built around learning and transmitting knowledge to succeeding gene- 
rations. "Evaluation culture" in Brazil is presently understood as an overall com- 
prehension and acceptance in the realm of institutions, organizations and society 
in general, of evaluation importance, need, practice and utilization. In other wor- 
ds, when "evaluation culture" is present, there is a set of learned beliefs, values, 
styles and behaviors that makes evaluation welcome. Evaluation here does not 
threaten or condemn. 

On the other hand, understanding policy as a deliberate plan of action to 
guide decisions and achieve rational outcome(s) "evaluation policy" is perceived as 
a set of guidelines that establishes rules and procedures to properly conduct 
planning, implementation and utilization of evaluation, in all circumstances or 
levels of possible implementation.. However, such guidelines must be on one 
hand, directive enough to ensure consensus that is essential to achieve air-com- 
mon major purposes in a disciplined way. On the other hand, they must be flexi- 
ble enough to respect diversity and pluralities of interests among the various 
settings of evaluation conception and practice. Evaluation policies should be well 
documented and be transparent for both evaluator and evaluee. An "evaluation 
policy" must answer questions such as: "What makes up the evaluation? Who is 
responsible for what? What standards do we use? Thus an "evaluation policy" 
should contain: a clear explanation of the concept and role of evaluation; the 
adopted Evaluation Standards; clear definition of roles and responsibilities; em- 
phasis on the need to operationalize the policy through evaluation guidelines; 
how evaluations are planned and implemented; mechanisms for the follow-up of 
evaluations; and the dissemination procedures for evaluations 

One single country in different evaluation stages 

As said before evaluations of diverse natures were produced and these can be 
grouped into three categories: (1) evaluation without a guiding policy; (2) evaluation 
with a pre-determined policy designed by a small group, not submitted for wide and 
substantial discussion, nor using language accessible to specific groups, often lacking 
transparency and wide dissemination; (3) evaluation with a pre-determined policy 
designed in an open and collective manner, and widely disseminated. 
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It is important to point out that our experiences showed that the further one 
develops an evaluation culture the closer one approaches the third type of evalua- 
tion practice mentioned above. Nevertheless, in situations where evaluation culture 
is not yet fully developed, building evaluation policies in a collective way has been 
successful in promoting integration and commitment among stakeholders. This 
result represents, indeed, a starting point to develop an evaluation culture. It is, 
indeed, one way of improving the process of "sowing the seeds". 

Especially in the educational and social areas where many evaluation systems 
are implemented by the Federal Government, Brazil has largely expanded its educa- 
tional policy development efforts. 

The significance of Evaluation Policy 
for evaluation and meta-evaluation 

Evaluation policy is meaningful to evaluation and meta-evaluation in the sense of 
ensuring that everybody would have the same understanding about evaluation, its 
purpose, and the way it must be conducted. Moreover, policy is, in general, a formal 
written statement that is useful to guide and determine how an evaluation should be 
implemented so that it is possible to establish a clear limit, avoiding ethical problems. 

When policies are not clearly stated evaluation borders become nebulous and 
easily crossed. Consequently, evaluators are asked to answer questions that are not 
within the scope of the evaluation and evaluees are judged about issues that do not 
belong to the content of the evaluation. 

Although Brazil has advanced regarding evaluation and evaluation policy, it still 
lags behind in including meta-evaluation in such policy. Evaluation policy is in fact 
useful to orient evaluation users to conduct meta-evaluation, which is the best way 
one knows to guarantee the quality of the evaluation process. In this sense, it is 
worthwhile mentioning that in spite of having a long way to go in reaching the 
highest expectations about meta-evaluation, there have been some remarkable efforts 
in the Brazilian evaluation scenario in terms of training evaluation users themselves 
as evaluation critics and evaluation partners. 

However, training ©f professionals for the practice of evaluation as well -expan- 
ding knowledge and experience in the field by reaching all those involved and interes- 
ted in developing and improving the the quality of the evaluand is still a challenge. 

Possible consequences of lack of policy 

Evaluation is usually a challenge to both evaluation and the evaluee. As mentioned 
before, presence of an evaluation culture facilitates facing this challenge; its absence 
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frequently generates fear of evaluation. Although it is hard to frame an evaluation 
policy in a context of poor evaluation culture, there should necessarily be an effort 
from evaluators towards constructing a policy. The evaluation policy does not specify 
the steps that must be followed but does guide how to conduct evaluations. Conflicts 
in the evaluative process emerge either when principles and considerations are not 
clear to all involved or when they are violated. Briefly, these conflicts appear in the 
form of evaluations of deteriorated quality, failure in identifying and involving stakehol- 
ders, and inadequate training of the evaluator, or results not fully utilized. Conduc- 
ting an evaluation without evaluation policy, in a field without evaluation culture, 
provokes chaos. Evaluation then starts working without directions, and evaluees give 
information without any commitment. When this happens it is important to ask why 
such evaluation is being conducted. If evaluators are not sure about why, where, and 
when they want to go, and evaluees do not know why, how, and what they must 
answer, it is important to ask: Is it really worthwhile to conduct this evaluation? Will 
someone use its results? Is it possible to trust the results? Is it fair dedicating human 
and financial efforts to this process? 

All these questions remind us of a statement by Michael Scriven (2004): 

In my view, one of the most important questions professi- 
onal evaluators should regularly consider is the extent to 
which evaluation has made a contribution to the welfare of 
humankind and, more generally, to the welfare of the pla- 
net we inhabit - and, while we're at it, to the welfare of the 
other celestial bodies we are beginning to invade. 

Brazilian experiences about culture-policy-practice 
inter-relations in evaluation: a success case 

The Program 

Brazil is a country seriously affected by social inequalities. Further, the social 
scenario is so complex that proposed solutions could be completely inappropriate 
as the apparent cues of poverty could be disguising deeper aspects that are actually 
causing the problem deserving attention. If these elements are not properly tack- 
led, the real problem may continue and probably become worse. In this sense, a 
sound evaluation is needed to uncover what is actually happening instead of what 
we would like to happen (PATTON, 2005). Although ambitious, there is reason to 
believe that evaluation can construct a new reality as long as it is present in the 
different sectors of society and it counts on multidisciplinary teams of competent 
professionals. The case we describe here refers to the joint action of a community 
development program and its evaluation in a slum area of Rio de Janeiro, Brazil, 
with a population of approximately 2.000 people. This program named "Betting on 
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the Future" is sponsored by Cesgranrio Foundation, a non-profit organization 
located near the target community, with a mission to promote and innovate evalu- 
ation theory, methodology and practice. 

This specific program was developed in the context of the city of Rio de 
Janeiro, and addressed an area that includes four extremely poor communities: 
Parque Andre Reboucas, Vila Santa Alexandrina, Paula Ramos e Escadaria. The 
main purpose of this initiative was to contribute to the social and economical 
development of those communities, aimed at improving their quality of life. Two 
important questions were initially raised. The first one relates to our awareness of 
the information (or lack of it) concerning the target population. The second relates 
to the identification of priorities by the target population. A situation analysis was 
conducted to answer these questions. This generated an overall view of each com- 
munity in terms of their needs, growth opportunities, realizations and limitations. 
Through this it was possible to detect basic information such as the number of 
people in each community, their home status, years of schooling, occupations and 
professions, as well as specific information about the needs of different segments 
in each community. The results of this analysis prompted the development of a 
program called "Betting on the Future". It aims at promoting sustained gains con- 
cerning the quality of life of the communities. To guarantee the success of the 
program both in terms of attaining objectives and sustaining results, an evaluation 
process was designed. Additionally, to ensure the excellence of such evaluation 
process, a meta-evaluation procedure was also planned. 

The unique aspects of this experience were: (a) giving due recognition to the 
community's knowledge for detecting its own needs; (b) selecting the best part- 
nership to develop the activities and a competent team to conduct evaluation and 
meta-evaluation; (c) emphasis on community capacity building in evaluation, self- 
determination, and self-sustainability, and continuous feedback to partners, spon- 
sors and the community. 

The birth of an evaluation 
policy attached to a social policy 

The Brazilian sociopolitical context as mentioned earlier requires that any 
form of evaluation intervention be integrated with social environment , if one 
intends to reach success. 

Simultaneous to the identification of priorities, Cesgranrio Foundation de- 
signed a social policy and an evaluation policy as it commissioned an external 
evaluation for the Program. A fundamental assumption of the social policy 
was is- the belief that social programs should aim at social promotion instead of 
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a mere non empowering assistance. In this sense such programs should incor- 
porate planning, coordination, monitoring and evaluation to avoid duplication 
of efforts or lost resources, in order to guarantee a meaningful change in the 
social context. Furthermore, a social policy would concretely be applied throu- 
gh properly evaluated projects and actions. Thus, with the purpose of consoli- 
dating its intervention in the social area, Cesgranrio Foundation designed and 
implemented a social policy to guide its action in such area. The evaluation 
process had to be accurate and timely, but subtle and sensitive, approaching 
the right people at the right time making sure no harm was done either to the 
evaluation team or the community. Fortunately, the commitment and enthu- 
siasm of those involved in the evaluation process overcame these limitations. In 
addition, the team tried to integrate different evaluation models focused around 
a responsive approach (STAKE, 2004). Our purpose was to have an external view 
on the overall Program in order to portray the quality as well as the possible 
limitations and the anxieties and opinions of the beneficiaries. Thus, the evalu- 
ation reproduced the peculiarities of the program, demonstrating a case of an 
evaluation historian (CRONBACH, 1980) where we describe, register and inter- 
pret the singular day-to-day process of each focused scenario. During its deve- 
lopment, the evaluation considered the values, concerns, and perceptions of 
the program stakeholders and strived to identify the program's merit, i.e., its 
internal value, and also to discover its relevance in the success demonstrated by 
its results, repercussion and possible impacts. 

Since it was an external evaluation, the evaluation team was composed of pro- 
fessionals with no direct connection with the Program. The advantage gained by 
this procedure was that we could offer a vision complementary to the self-percep- 
tion of the program stakeholders. Both internal and external perspectives were 
necessary to achieve the desired success of the evaluation. The approach used in the 
external evaluation was essentially inclusive (MERTENS, 2003) because it emphasi- 
zed the insertion and participation of all communities affected by the decisions 
based on the evaluation process. 

This approach was present during all steps of the evaluative process. In this 
context, the role of the evaluator was a proactive one in the sense of intervening to 
change the elements that favor social injustice. 

The starting point of the evaluation was the following evaluative question: 

- To what extent is the project being developed efficiently? 
(Merit) 

- What are the evidences of impact on beneficiaries and 
communities at large in regards to the improvement of 
their quality of life? (Impact or Worth) 
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It was made sure that the identified merit and impact indicators were consistent 
with the adopted evaluation approach. We collected data from three different sources: 
(a) beneficiaries, (b) partners, and (c) community. Several data collection procedures 
were used: (a) checklists, (b) interviews, (c) unobtrusive measures, and (d) focus groups. 

A unique characteristic of the evaluation process has been that of offering 
continuous feedback to all the stakeholders of the program during the whole tra- 
jectory of its implementation. This procedure has undoubtedly prompted favorable 
changes among beneficiaries, partners and communities overall. 

Results 

An essential feature of the whole evaluative task was the continuous utilization 
of its process and results (PATTON, 1997). The present evaluation was constantly 
oriented towards such an effort. . The evaluation team understood that in the 
social service area more than in any other, there is a need for permanent follow-up 
to the development of actions and opportunities during the intervention of any 
program from its conception to its conclusion and repercussions. 

In this sense, it is hoped that the results of this program will affect not only the 
target communities but the overall Brazilian society, for the benefit of many, inclu- 
ding those who are responsible for conducting programs, for developing evaluative 
or research studies, or holding policy discussions in the social arena. 

Changes observed as evidences of merit and impact are summarized below. 

In relation to merit: 

• Quality of Program Actions: the results show that: 
(1) there is a persistent interest for the best possible 
implementation of the program activities in all the areas 
of action, and (2) an increase of competency among the 
professionals that work in the program. 

• Environmental conditions: there are indeed poor 
conditions in the environment over which the inhabi- 
tants have no power to intervene. These conditions ham- 
per the possibilities of a healthy life. This is the case of the 
sewer, The program, however, has been acting insistently 
and little by little some changes have been observed such 
as "not pulling leaves out of trees", "not mistreating ani- 
mals", "not throwing rocks" etc. Children and adoles- 
cents already show these behaviors with great enthusi- 
asm and conviction. In school the improvement is rema- 
rkable. Thus, from school to community, they are gro- 
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wing. 

• Diversified activities: the results show a constant ad- 
justment of program activities in the sense of responding 
to the interests of the beneficiaries and rapidly resolving 
conflicts. There are also evidences of the integration of 
actions between projects. This has been ensuring the qua- 
lity of results and at the same becoming a model for the 
beneficiaries and the community. 

In relation to impact: 

• Personal development: a remarkable result is the evi- 
dence of respect for moral values that have been emer- 
ging little by little. The program has showed impact on 
solidarity which, already existent, has been further rein- 
forced. An improved relationship among the inhabitants 
has been noted. Mothers are more involved in the pro- 
gram and fathers, little by little. Self esteem has been 
enhanced. 

• Vulnerability: results show that children and adoles- 
cents still work but only to collaborate at home. Violence 
against children and youngsters has been considerably 
reduced. 

• Health: the strongest evidence of change was the re- 
duction of the answer "never" when the beneficiaries were 
inquired about (a) men forty years or older submitting 
themselves to prostate examination and (b) women sub- 
mitting themselves to preventive cancer examination. A 
greater care towards the hygienic conditions of the com- 
munity physical space was also observed. 

• Education: the results point etrt to visible improvement 
in school performance. A great increase in interest for 
reading and writing was evident. School dropout was 
drastically reduced. 

However, the most important asset of this whole experience was the support of 
the evaluation practice by an evaluation policy, which was itself attached to a social 
policy. From now on, all the evaluation interventions should be in harmony with the 
evaluation policy. This approach has been an irreplaceable coach to develop the 
needed evaluation culture in the target community. 

Conclusions and recommendations 

In synthesis, collectively constructing an evaluation policy prevents misunders- 
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tandings, conflicts, and avoids inappropriate use of results, in other words, it avoids 
evaluation pain. Moreover, evaluation culture emerges from the positive impact of 
the evaluation process, which must respect the values of the community or society 
involved. In this sense, our recommendations for those who practice, study or 
make decisions in relation to evaluation, and even more to the evaluees, are: 

• Evaluation culture, policy and practice should work together; 

• Evaluation policy has to be a public document; 

• Evaluation policies have to be constructed in a participa- 
tive way; 

• Evaluation policies are a necessary guide to true evaluations; 

• Evaluation should not be developed in a community just 
because evaluators want to do it but only and solely when 
the community understands its importance and takes it 
on its own. 

Finally, to paraphrase what as Patton (1997) wisely said once, evaluation is too 
important to be left to evaluators alone. 
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